Metrics
Session Room: Table 3 Time Slot: 2pm on Saturday Organizer: Jeff Potts Note-taker: Richard Esplin Metrics: * Forum posts / list posts (user accounts doesn't help any except how to detect spam and ant-spam effectiveness), intervals between posts, broken down by thread * Claimed commits * Wiki edits * Unique visits, pages per visits * Social network growth, twitter engagement * "Golden Motion" / desired action: download * User feedback button within product that collects software stats * Referrals * Who signs CLAs. * Number of times your technology is in a job description (especially vs an established competitor) Tools: * Google Analytics * Sentiment analysis (but it seems the tools are lacking, especially the open source ones) * Birt * Queries published by LibreOffice and Meego * Chart-io, Saiku * Metrics-wg @ The Open Source Way * Excel * Drupal Module system is a good example * Debian popularity contest Take-aways: * Looking at time periods / intervals of engagement and averages are usually more insightful than raw numbers. Trends. * Number of new threads over time can be a good look at community health. * Interesting differences between organic growth and forced growth * Interesting difference between internal and external communities. Internal communities are easier to measure. * Analytics are tailored based on who will be analyzing / using the data * Dr. Michael Wu (Lithium) has good material online for gaging the health of a community (99 / 1 rule, Lorenze Curves) * It is easy to get blips. Long term trends are what matter. (Months, not weeks or years.) If you see a spike, it is worth the time to identify what happened. * Web metrics vs tool analytics * Different purposes: Prove worth to boss, eliminate barriers to engagement (apply a sales funnel mindset to community metrics), detect problems * We need to measure the objectives of why people are participating. * Living in the community is an import way to color what you pull out of the statistics. Sometimes you need to go and look. * Growth is not the only measure of health. Growth is quantitative, but health is qualitative. * The meaning of metrics will change over the lifetime of the community, and sometimes it needs to be re-evaluated what needs to be measured. * What you measure is what you get. Measuring the wrong the wrong thing can create the wrong incentives. * Some times you need to measure things that don't really matter because of business concerns (upper management). That is a business problem that needs to be addressed. (Forum posts should go down as documentation quality goes up.) * What you measure should be a derivative of preset goals. But if you aren't sure what your goals are, it can work to just measure as much as possible. Lots of data can help with gamification, and big-data analysis can help uncover hidden connections. * We all have lots of platforms, and you need to call out to all of them to get the metrics * Pick tools with good API's * Open source software communities are a bit of a niche. Communities that involve people in physical space that do things would have some different metrics, though they will be similar. Process: * Define goals first * Gather metrics * Make decisions * Gamify Questions: * How to tell what metrics we are measuring are signal or noise? How to gage metric relevance? What are the effective metrics? * How to measure health vs growth vs activity? ** Health is a reputational judgment about community. Activity is easy to measure. ** The forums that have the most momentum are usually the ones that are the most positive * How to connect a metric to an action? * What actions should we reward people for? * How to use metrics to determine when there is a problem? Action Items: * Megan: 10Gen / Mongo Metrics Framework slide (genericized), or lead session on Sunday * Ross Turk: Mind Map of metrics that would be nice to gather (posted to twitter with #cls12)